Adding augmented reality to a sub-view of a high resolution central video feed

ABSTRACT

Techniques are disclosed to add augmented reality to a sub-view of a high resolution central video feed. In various embodiments, a central video feed is received from a first camera on a first recurring basis and time-stamped position information is received from a tracking system on a second recurring basis. The central video feed is calibrated against a spatial region encompassed by the central video feed. The received time-stamped position information and a determined plurality of tiles associated with at least one frame of the central video feed are used to define a first sub-view of the central video feed. The first sub-view and a homography defining placement of augmented reality elements on the at least one frame of the central video feed are provided as output to a device configured to use the first sub-view and the homography display the first sub-view.

CROSS REFERENCE TO OTHER APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/381,877 entitled ADDING AUGMENTED REALITY TO A SUB-VIEW OF A HIGHRESOLUTION CENTRAL VIDEO FEED filed Jul. 21, 2021 which is incorporatedherein by reference for all purposes.

BACKGROUND OF THE INVENTION

Conventional camera tracking systems typically track an object viamanual operation of a camera focused on an object of interest or ananalysis of the subject matter that is captured by each respectivecamera. In a first typical method, an object is tracked manually by acamera operator, who ensures that the object is always within a frame ofview. In a second typical method, a series of images is captured by acamera, and these images are analyzed to determine opticalcharacteristics of an object that is tracked, such as identifying acolor associated with the object or a silhouette of the object. Theseoptical characteristics are recognized in further images, allowing theobject to be tracked through the progression of the series of images.

In the first example method, numerous resources such as equipment andcamera operators are required to effectively track different types ofobjects. In the second example method, the conventional systems aresusceptible to losing track of the object if the object quickly dartsout of a line of sight of the camera or if there are multiple objects inthe line of sight of the camera that are optically similar to thedesired object. Both example methods are typically not tailored toindividual viewer interests and are more commonly found in broadcastmedia for a general audience. Thus, there is a need for improved objecttracking and display systems.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 is a flow diagram illustrating an embodiment of a process foradding augmented reality to a sub-view of a high resolution centralvideo feed.

FIG. 2A shows an example of a sub-view obtained in various embodimentsof the present disclosure.

FIG. 2B shows an example of a sub-view at a first zoom level obtained invarious embodiments of the present disclosure.

FIG. 2C shows an example of a sub-view at a second zoom level obtainedin various embodiments of the present disclosure.

FIG. 2D shows an example of a sub-view at a third zoom level obtained invarious embodiments of the present disclosure.

FIG. 3 is a flow diagram illustrating an embodiment of a process forcompositing a sub-view including augmented reality.

FIG. 4A shows an example of components for constructing a compositeimage according to various embodiments of the present disclosure.

FIG. 4B shows an example of a composite image obtained by combiningcomponents according to various embodiments of the present disclosure.

FIG. 5 is a block diagram illustrating an embodiment of a system foradding augmented reality to a sub-view of a high resolution centralvideo feed.

FIG. 6 shows an example of a graphical user interface for addingaugmented reality to a sub-view of a high resolution central video feedaccording to an embodiment of the present disclosure.

FIG. 7 shows an example environment including a field of play thatincludes components of a tracking according to an embodiment of thepresent disclosure.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a processor configured to execute instructions storedon and/or provided by a memory coupled to the processor. In thisspecification, these implementations, or any other form that theinvention may take, may be referred to as techniques. In general, theorder of the steps of disclosed processes may be altered within thescope of the invention. Unless stated otherwise, a component such as aprocessor or a memory described as being configured to perform a taskmay be implemented as a general component that is temporarily configuredto perform the task at a given time or a specific component that ismanufactured to perform the task. As used herein, the term ‘processor’refers to one or more devices, circuits, and/or processing coresconfigured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications, andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

Techniques for adding augmented reality to a sub-view of a highresolution central video feed are disclosed. In various embodiments, ahigh resolution central video feed is captured by a high resolutioncamera having a full view of a competition space. The high resolutioncentral video feed can be partitioned into one or more sub-views. Invarious embodiments, a sub-view is an isolation shot of a portion of thecompetition space and can focus on one or more objects of interest suchas a ball, player, or group of players. Using the example of a gridironor North American football game, the high resolution central video feedcaptures at least the entire playing field. In various embodiments, thehigh resolution central video feed also captures areas along the side ofthe field of interest such as the bench or other an area where playersor other objects of interest might be while not active on the field. Theisolation shot can follow the ball, a specific player, a group ofplayers, and/or other objects of interest. Augmented reality can beadded to the isolation shot to enhance the viewing experience of aspectator of the football game (sometimes also called a user of thedisclosed system). The augmented reality can more clearly show scrimmagelines, statistics associated with something that is happening during thegame (such as the speed of a ball, shot percentage), advertisingcontent, etc. Unlike conventional augmented reality components that getadded to a sports event in a television broadcast, the augmented realitycomponents added to the isolation shots according to the disclosedtechniques can be tailored to the content of the isolation shots and/orthe interests of a user. The augmented reality can be added either by acentral server and then distributed to client devices, or locally by aclient device using metadata transmitted by a central server.

The examples herein primarily use a gridiron or North American footballgame, but this is merely exemplary and not intended to be limiting asthe techniques can be applied to a variety of sporting (andnon-sporting) events that would benefit from capturing video of an areaand then providing augmented reality for sub-views of the area.

The disclosed techniques find application in a variety of settingsincluding, but not limited to: quickly and easily changing shots,defining a shot until an event happens to an object of interest (e.g., aball is caught), and then following the object of interest (e.g., aplayer in possession of the ball). This improves the efficiency ofcapturing sporting events by complementing or replacing at least some ofthe traditional methods of capturing video of sporting events.

FIG. 1 is a flow diagram illustrating an embodiment of a process foradding augmented reality to a sub-view of a high resolution centralvideo feed. This process may be implemented on a system such as the oneshown in FIG. 5 . In various embodiments, the process is implemented byisolation shot engine 514 in cooperation with augmented reality engine512.

The process begins by receiving, on a first recurring basis, atransmission of a central video feed from a first camera (100). Thecentral video feed refers to a series of one or more frames of videocapturing a full view of a competition space. In other words, thecentral video feed encompasses an entire field of play and captures anentire scene. In some embodiments, the central video feed also capturesareas adjacent to or associated with the competition space such as thebench or other an area where players or other objects of interest mightbe while not active on the field.

Referring to FIG. 7 , a central video feed is received from a firstcamera 740, for example. In some embodiments, the camera 740 is a fixedcamera (e.g., the camera is limited in movement in at least one axis).For instance, in some embodiments the camera 740 is fixed such that thecamera is capable of having a variable tilt, pan, and/or zoom, but isnot able to be physically moved to another location. In someembodiments, the camera 750 is fixed such that the camera is not capableof moving in any axis and/or zoom. In some embodiments, the cameras arenot fixed and field registration data can be used to obtain theorientation to apply an isolation camera effect.

The camera can be disposed in a variety of locations and orientationssuch as at a first end portion (e.g., a half court line, a 50-yard line)of a field of play in a landscape orientation or at a second end portion(e.g., an end zone, a goal) of a field of in a portrait orientation,among others. As further described herein, the camera may be configuredto capture high resolution imagery (a high resolution central videofeed) such that when partitioned into sub-views, the sub-views are ofsufficient resolution to be displayed on a user's device such assmartphone. The camera 740 is in communication with a network in orderto communicate with one or more devices and systems of the presentdisclosure.

In some embodiments, the central video feed comprises and/or is includedin a plurality of central or other video feeds, each generated by one ormore cameras located and oriented to generate video of at least aportion of the field of play. In some embodiments, the central videofeed and/or another video feed may be generated at least in part bycombining video data generated by a plurality of cameras, such as acomposite or otherwise merged or combined video. In some embodiments, aplurality of cameras can be provided around the environment, each ofwhich encompasses an entire field of play. Several central video feedsmay be generated to allow for different angles and perspectives of thescene of play. A central video feed may include timestamps from eachdata stream to calibrate/coordinate the composition of scenes.

The process receives, on a second recurring basis, a respectivetime-stamped position information from a tracking system (102). Invarious embodiments, the time-stamped position information correspondsto a player or other object of interest. For example, a player can be apoint in space and the amount of padding around the point defines alevel of zoom around that player as further described herein.

Any tracking system may be used, and the one described for this exampleis merely exemplary and not intended to be limiting. In variousembodiments, a tracking system includes tracking devices worn by acorresponding subject (e.g., players) participating in a competition inthe spatial region or associated with other objects of interest (e.g., aball). Respective time-stamped position information from each trackingdevice is received. For example, each tracking device transmitspositional information that describes a time-stamped position of thecorresponding subject in the spatial region.

An example of a tracking system and transmission of time-stampedposition information is further described with respect to FIG. 7 .Referring to FIG. 7 , in one exemplary tracking system, an array ofanchor devices 720 (e.g., anchor device 720-1, anchor device 720-2, . .. , anchor device 720-Q) receives telemetry data from one or moretracking devices associated with a respective subject or object ofinterest of the game. Subjects or objects of interest (represented bysquare and circles) may have one or more tracking devices attached totheir body or otherwise monitoring their movements/behavior.

The process calibrates the central video feed against a spatial regionencompassed by the central video feed (104). The process performs thecalibration including by: partitioning at least one frame of the centralvideo feed into a plurality of tiles at a first resolution, anddetermining a homography defining placement of augmented realityelements on the at least one frame of the central video feed.

The central video feed is calibrated against a spatial regionrepresented in at least two dimensions that is encompassed by thecentral video feed. In some embodiments, the spatial region is a regionthat is captured by an array of anchor devices 720. The spatial regioncan be a field of play of a live sports event. In some embodiments, thecalibration of the central video feed includes determining an equivalentportion of the central video feed for a coordinate system used by thepositional information (e.g., telemetry data). Since standard fields ofplay of competitive sports include boundary lines of regulation, uniformlength and thickness/widths (e.g., an out of bounds line, a half courtline, a yard line, etc.), these lengths and thicknesses can be used todetermine coordinate positions in the video feed. For instance, if aline on a field of play is known to have a uniform thickness (e.g., 6centimeters thickness) and the thickness of the line in the centralvideo feed is determined to reduce linearly from a first thickness to asecond thickness, then an exact location of a subject with respect tothe line can be determined in the central video feed.

In various embodiments, the process partitions at least one frame of thecentral video feed into a plurality of tiles at a first resolution.Referring briefly to FIG. 5 , the central video feed, which captures anentire field of view is partitioned into nine tiles (labeled isolationshots in the figure). Each of the tiles captures a sub-view of theentire field of view. The number of tiles is merely exemplary and notintended to be limiting. Other examples include dividing an 8K frame ofthe high resolution camera feed into 16 (or more generally, n) segments,depending on a desired resolution. In various embodiments, the tiles arestored in a server, and a client requests a tile or subject/object ofinterest and the corresponding tile is delivered to the client. Anexample of partitioning at least one frame of the central video feedinto a plurality of tiles at a first resolution is further describedwith respect to FIG. 5 .

In various embodiments, camera calibration information can be used todetermine a homography defining placement of augmented reality elementson at least one frame of the central video feed. Referring briefly toFIG. 5 , AR engine 512 determines a homography and isolation shot engine514 uses the determined homography to output information to clientdevice 550. A client can use homography data or metadata to render animage with augmented reality elements incorporated into the image. Anexample of a process to use homography to place augmented realityelements on frame(s) of a central video feed is further described withrespect to FIG. 3 .

The process uses the received time-stamped position information and theplurality of tiles associated with at least one frame of the centralvideo feed to define a first sub-view of the central video feed (106).In various embodiments, the first sub-view includes a subset of theplurality of tiles associated with at least one frame of the centralvideo feed. The first sub-view is associated with a first set of one ormore subjects. For example, the first sub-view may be associated with(include/show) a first subject of the plurality of subjects in thespatial region. The first sub-view comprises, for each of a plurality offrames comprising the central video feed, a corresponding sub-frameassociated with the first set of subject(s). By way of non-limitingexample, a sub-view defines a frame around a single player or multipleplayers, can follow an object of interest such as a ball, can follow oneor more players at all times even when the player is not in the field ofplay (e.g., the player is on the bench), or can follow othersubjects/objects of interest such as game officials. A player or objectof interest can be followed by using metadata. The metadata is used tocreate an isolation shot on an on-going basis that creates the visualeffect of a camera following the player or object of interest.

For example, in some embodiments, the process applies to each of aplurality of sequential frames of video data a mathematicaltransformation that is based at least in part on correspondingcamera/video calibration data to determine, based on timestamp datacomprising the received positional information and the positionalinformation associated with each timestamp (e.g., XYZ coordinates of asubject A), a subset or portion of each sequential frame that isassociated with the corresponding positional information of the subjectA. The determined subsets/portions of the sequential frames are used toprovide a sub-view of the central video feed associated with the subjectA.

The sub-view is a different resolution from the central video feed invarious embodiments. Despite being a different resolution, the qualitydifference is not necessarily noticeable to the average spectator sothat the viewing experience remains enjoyable. For example, the centralvideo feed is provided at a first resolution (e.g., a native resolutionof the camera 140) such as between 2K and 12K. To this point, in someembodiments the central video feed includes a plurality of fulltwo-dimensional frames (e.g., a first frame associated with a first timepoint, a second frame associated with a second time point, . . . , a nthframe associated with an nth time point). Each respective fulltwo-dimensional frame in the plurality of full two-dimensional frameshas a first dimension size and a second dimension size (e.g., ahorizontal size and a vertical size such as a number of horizontalpixels and a number of vertical pixels). The first sub-view includes acorresponding sub-frame, for each respective full two-dimensional framein the plurality of full two-dimensional frames. Each correspondingsub-frame is a portion of a corresponding full frame (e.g.,sub-view/isolation shot 1 and sub-view/isolation shot 2 of FIG. 5illustrates instantaneous sub-frames of the central video feed fullframe (entire field view) of FIG. 5 ).

As described herein, a first sub-view of the central video feed can bedefined at a second resolution that is less than the first resolution(the resolution of the tiles). For instance, the first resolution is atleast four times, six times, or eight times the pixel resolution of asecond resolution of a video that is partitioned from the central videofeed.

A sub-view can have varying levels of zoom. The zoom can be definedaround a player or object of interest by setting padding around theplayer as further described with respect to FIGS. 2A-2D.

In some embodiments, each sub-frame has a third dimension size and afourth dimension size. Moreover, the third dimension size can be a fixedfraction of the first dimension size and the fourth dimension size is afixed fraction of the second dimension size. For instance, the fixedfraction of the first dimension size and the fixed fraction of thesecond dimension size of a same fraction (e.g., 10%, 20%, 30%, . . . ,90%). Similarly, the fixed fraction of the first dimension size can be afirst fraction and the fixed fraction of the second dimension size canbe a second fraction different than the first fraction (e.g., thecentral video feed is captured in a landscape orientation and eachsub-view is partitioned in a portrait orientation). By way ofnon-limiting example, (i) the first dimension size is 7680 pixels andthe third dimension size is 3840 pixels, and the second dimension sizeis 4320 pixels and the fourth dimension size is 2160 pixels; or (ii) thefirst dimension size is 8192 pixels and the third dimension size is 3840pixels, and the second dimension size is 4320 pixels and the fourthdimension size is 2160 pixels. In some embodiments each respective fulltwo-dimensional frame in the plurality of full two-dimensional framesincludes at least 10 megapixels to 40. In some embodiments a sub-view(e.g., the first sub-view) includes a corresponding sub-frame, for eachrespective full two-dimensional frame in the plurality of fulltwo-dimensional frames, that includes less than 5 megapixels to 15megapixels.

The coordinates of a center of the first sub-view within the centralvideo feed changes over time, without human intervention, in accordancewith a change over time in the position of the first subject asdetermined from recurring instances of the receiving that occur on thesecond recurring basis by the overlapping. In some embodiments, thecenter of the first sub-view is associated with position coordinates(e.g., XYZ) generated by a tracking device worn or otherwise associatedwith the subject. In some embodiments, a subject may wear multipletracking devices and the first sub-view is centered based on a set ofcoordinates generated based on tracking data from the plurality ofdevices. For example, device data from multiple tracking devices worn bya subject may be correlated, e.g., based on timestamp data, and ageometric or other center set of coordinates may be computed based onthe coordinates generated by the respective tracking devices.

In some embodiments, the first sub-view of the central video feed iscommunicated to a remote device (e.g., client device 550 of FIG. 5 )independent of the central video feed. Accordingly, the communicatingcauses the remote device to display the first sub-view of the centralvideo feed. By way of non-limiting example, the remote device is ahandheld device such as a smart phone, a tablet, a gaming console, afixed computer system such as a personal home computer, or the like.Moreover, the communicating can occur wirelessly (e.g., over a network).

In various embodiments, at least a first subject in the subset ofsubjects is selected. The selection of the at least first subject can beconducted via a computer system for example by an operator of thecomputer system (e.g., a video production specialist, a producer, adirector, etc.), an end user of each respective remote device (e.g., viaa respective user device 550), or automatically. For example, a firstsubject is selected automatically based at least in part on proximity(being within a threshold distance) to a ball or other subject (forexample, a previously selected subject with which the subject isassociated, such as in a one-on-one match or by being associated withcomplementary positions, such as opposing offensive and defensivelinemen). Moreover, a sub-view may be selected from a wider collectionof sub-views (e.g., a list of available sub-views, a preview ofavailable sub-view, etc.). The wider collection of sub-views includes asub-view for each player active in a competitive game (e.g., twenty-twosub-views for an American football game). This end-user selection allowsfor each user to select one or more subjects according to their desire.For instance, if the end-user has a list of favorite subjects spreadacross multiple teams, the end-user may view sub-views of each of thesefavorite subjects on a single remote device and/or display.

In some embodiments, an identity of the first subject is received at theremote device. For instance, the first sub view includes informationrelated to the identity of the first subject (e.g., a name of the firstsubject). This identity of the respective subject allows for an end-userto quickly identify different sub views when viewing more than one subview. In some embodiments, a tracking device is attached to (e.g.,embedded within) a ball that is being used in the competitive sport onthe spatial region. Accordingly, the identity of the first subject isdetermined, without human intervention, based on a determination ofwhich subject in the plurality of subjects is currently closet to theball using the respective transmission of time-stamped positionalinformation from each tracking device.

The process outputs the first sub-view and the determined homography toa device configured to use the first sub-view and the homography displaythe first sub-view (108). In various embodiments, the client device usesa blank scene (empty field), player information (e.g., trackinginformation), and AR component information (the determined homography)to create a composite image showing the combination of the players onthe field along with AR components. An example process for displayingthe first sub-view including AR components is further described withrespect to FIGS. 3 and 4 .

In various embodiments, one or more steps of the process of FIG. 1occurs during a live game in which the plurality of subjects isparticipating. However, the present disclosure is not limited thereto.For instance, the communicating can occur after a live game (e.g., suchas viewing highlights of the live game or a replay of the live game).

FIG. 2A shows an example of a sub-view obtained in various embodimentsof the present disclosure. For context, the entire field is shown indashed lines in FIG. 2A. A sub-view (labeled isolation shot) isdesignated by the box around a subset of the players on the field. Inthis example, the object of interest is the player represented by thecircle in the center of the isolation shot.

FIG. 2B shows an example of a sub-view at a first zoom level obtained invarious embodiments of the present disclosure. The zoom is centered onthe object of interest labeled in FIG. 2A. The object of interest is apoint (defined by coordinates x, y or x, y, z) and the sub-view iscentered on the point in various embodiments. The level of zoom isdefined by the amount of padding surrounding the object of interest,here x″.

FIG. 2C shows an example of a sub-view at a second zoom level obtainedin various embodiments of the present disclosure. Compared with FIG. 2B,the sub-view here is more zoomed out so that the players looksmaller/less detailed. Here, the padding around the object of interestis a different value (x) from the padding in FIG. 2B, which causes thezoom level to look different.

FIG. 2D shows an example of a sub-view at a third zoom level obtained invarious embodiments of the present disclosure. Compared with FIG. 2C,the sub-view here is more zoomed out so that the players looksmaller/less detailed. Here, the padding around the object of interestis a different value (x′) from the padding in FIG. 2C, which causes thezoom level to look different.

FIG. 3 is a flow diagram illustrating an embodiment of a process forcompositing a sub-view including augmented reality. The process can beperformed by the system of FIG. 5 . The process of FIG. 3 will beexplained using FIGS. 4A and 4B.

FIG. 4A shows an example of components for constructing a compositeimage according to various embodiments of the present disclosure. Thecomponents include a blank scene 400, players 420, and augmented realitycomponents 430. The blank scene 400 shows the field of play without anyplayers on the field. For example, the players 420 can be determined bysubtracting or otherwise removing the blank scene 400 from a framecapturing a scene of the players on the field (which may also usetracking data as further described herein). The AR components 430include any component that augments a frame of video. In this example,the AR components is scrimmage line 432. It can be displayed in avisually distinguished way such as highlighted color to help a user moreclearly see the scrimmage line.

FIG. 4B shows an example of a composite image obtained by combiningcomponents according to various embodiments of the present disclosure.This composite image is obtained by combining the blank scene 400,players 420, and AR components 430.

Returning to FIG. 3 , the process begins by capturing a blank scene(300). The blank scene 400 can be captured by a camera such as camera740 prior to any players entering the field. The blank scene can be abaseline or reference frame for other frames such as frames of videoshowing various states of game play.

The process determines augmented reality components to add to the scene(302). The augmented reality components can be determined based onpredetermined settings or user interests. For example, a scrimmage lineenhances the user experience of all users and thus this AR component canbe determined for all frames. The components can be determined based onthe positions of the players 420 by determining, based on tracking data,a location where the ball is placed after the end of the most recentplay and taking into account any penalty yards. Users may be interestedin other information such as statistics for a specific player. Thestatistics can be determined and added to the scene as an AR component,for example in the corner of the display. Referring briefly to FIG. 5 ,AR engine 512 is configured to determine the AR components in variousembodiments.

The process reconstructs an image by compositing the blank scene, thedetermined augmented reality components, and players (304). The processcreates a composite image by combining (superimposing, for example) theplayers 420 onto the blank scene 400 and then any AR components 430 ontop.

Referring briefly to FIG. 5 , client device 550 is configured to createa composite image in various embodiments. Alternatively, server 510 isconfigured to create a composite image and send the composite imageand/or related data to the client device 550.

FIG. 5 is a block diagram illustrating an embodiment of a system foradding augmented reality to a sub-view of a high resolution centralvideo feed. The system includes a server 510 and a client device 550.The client device 550 can be smartphone, computer, or other device onwhich one or more frames of video is rendered.

The server 510 includes AR engine 512 and isolation shot engine 514. Auser preferences store 516, configured to store user preferences and/orprofiles, may be provided locally on the server as shown or remotely. ARengine 512 is configured to determine one or more AR components to bedisplayed on frame(s) of video data. The AR components can be based onuser preferences or known preferences of an audience. For example, ascrimmage line is helpful when rendered to visualize a current state ofplay and can be determined as an AR component and added to a frame ofvideo. Other AR components can be more user-specific, depending oninterests of the user such as a user who is a fan of a specific playeror group of players. Isolation shot engine 514 is configured to performa process such as the one of FIG. 1 to determine a sub-view centered onplayer(s)/object(s) of interest.

FIG. 6 shows an example of a graphical user interface for addingaugmented reality to a sub-view of a high resolution central video feedaccording to an embodiment of the present disclosure. The graphical userinterface includes a plays panel 610, a video panel 650, and a playerpanel 680.

Plays panel 610 displays various plays associated with a sporting eventcurrently displayed in the video panel 650. The sporting event can beviewed live or after the event has concluded. Specific plays can beviewed by selecting a corresponding play in the plays panel. In thisexample, the user is viewing a specific play identified by play ID 195.Associated information is displayed such as the time of the beginning ofthe play and the state of the play including which player (if any) is inpossession of the ball. In some embodiments, the time of the beginningof the play and/or state of play information is determinedautomatically, e.g., by processing video content using artificialintelligence, machine learning, and/or related techniques. In someembodiments, the play start time and/or state of play information may bewholly or partly entered as input by a human worker.

Video panel 650 displays video of a sporting event. The video can be asub-view that is generated using the disclosed techniques. The sub-viewcan be a composite image including players and AR components. The videocan be maximized to fill an entire screen by selecting the icon on thebottom right of the video. There are also various options displayedalong the top of the video panel. In this example, the user can selectthe angle of the video feed. Here, the video is from an 8K camera on thehigh left of the field. There may be other video feeds available havingdifferent resolutions and/or in different positions around the field.Another option is the type of filter to apply to the video. In thisexample, the default is broadcast shading. Other filters include blackand white or other color schemes. The filtering can be performed locallyat a client device. The AR drop down menu enable a user to select one ormore AR components to be rendered on the video panel 650.

Although not shown here, an AR component such as a scrimmage line can bedisplayed in the video panel. The AR component can be customized to theuser. For example, the color of the scrimmage line can be according to apreference of the user. Unlike conventional scrimmage lines displayed ina television broadcast, a scrimmage line determined according totechniques of the present disclosure is more accurate because a viewport(making up the tiles, and can be thought of a virtual camera) is moved.

The locations and number of menus is merely exemplary and not intendedto be limiting. For example, the menus may instead be displayed on thesides or bottom of the video panel.

The player panel 680 shows a least a portion of the team rosters. Here,player 15, the quarterback for KC is highlighted because the user isinterested in this player. The video panel 650 is displaying a sub-viewcentered on player 15. The user can select one or more other players toview sub-views associated with the other players. The user can select“clear all” to reset customization/personalization.

FIG. 7 shows an example environment including a field of play thatincludes components of a tracking according to an embodiment of thepresent disclosure. The system is an example of one that can capture acentral video feed used at 100 and collect time-stamped positioninformation used at 102.

An environment 700 includes a field of play 702 in which a game isplayed (e.g., a football game). The environment 700 includes a region704 that includes the field of play 702 and an area immediatelysurrounding the field of play (e.g., an area that includes subjects notparticipating in the game such as subject 730-1). The environment 700includes an array of anchor devices 720 (e.g., anchor device 720-1,anchor device 720-2, . . . , anchor device 720-Q) that receive telemetrydata from one or more tracking devices associated with a respectivesubject of the game. As illustrated in FIG. 7 , in some embodiments thearray of anchor devices is in communication with a telemetry parsingsystem. Moreover, in some embodiments one or more cameras 740 captureimages and/or video of the sports event, which is used in forming thevirtual reproduction.

The camera 740 is a high resolution camera capable of capturing video ata high resolution such as 12K or other higher resolutions available onthe market. In various embodiments, the camera may have a variety oflens including those with less linear distortion, which is suitable fordividing the central video feed into sub-views and minimizing distortionin the sub-views. In various embodiments, the camera captures an imageof the field at a relatively high camera angle.

As described herein, the central video feed can be divided intosub-views, where padding defines a level of zoom of the sub-view.Because the central video is high resolution, the sub-views can bedisplayed on a user device at a varying levels of zoom without being toograiny. In FIG. 7 , square markers represent subjects a first team ofthe game while circular markers represents subjects of a second team ofthe game.

A respective transmission of time-stamped positional information (e.g.,telemetry data 230) is received from each tracking device 300 in aplurality of tracking devices. The recurring basis of receiving thetransmission of time-stamped positional information can be a ping rate(e.g., instantaneous ping rate 310 of FIG. 3 ) of a respective trackingdevice 300. In some embodiments, transmission of time-stamped positionalinformation from each tracking device in a plurality of tracking devicesoccurs at a bandwidth of greater than 500 MHz or a fractional bandwidthequal to or greater than 0.20. By way of non-limiting example, thetransmission of time-stamped positional information from each trackingdevice in a plurality of tracking devices is within 3.4 GHz to 10.6 GHz,each tracking device 300 in the plurality of tracking devices has asignal refresh rate of between 1 Hz and 60 Hz, and/or the recurringbasis is between 1 Hz and 60 Hz. Each tracking device 300 of theplurality of tracking devices sends a unique signal that is received bythe receiving, identifying a respective tracking device. Each trackingdevice can transmit biometric data (e.g., biometric telemetry 236)specific to a respective subject associated with the respective trackingdevice if biometric data is collected.

Each tracking device 300 is worn by a corresponding subject in aplurality of subjects that is participating in a competition on thespatial region. Further, each tracking device 300 transmits positionalinformation (e.g., telemetry data 230) that describes a time-stampedposition of the corresponding subject in the spatial region. In someembodiments, there are at least two tracking devices 300 worn by eachsubject in the plurality of subjects. Each additional tracking device300 associated with a corresponding subject reduces an amount of errorin predicting an actual location of the subject.

In some embodiments, the plurality of subjects includes a first team(e.g., a home team) and a second team (e.g., an away team). In someembodiments, the first team and/or the second are included in a leagueof teams (e.g., a football league, a basketball association, etc.). Thefirst team includes a first plurality of players (e.g. a first roster ofplayers) and the second team includes a second plurality of players(e.g., a second roster of players). Throughout various embodiments ofthe present disclosure, the first team and the second team are engagedin a competitive game (e.g., a live sport event), such as a footballgame or a basketball game. Accordingly, the spatial region is a field ofplay of the competitive game, such as a football field or a basketballcourt. In some embodiments, the subjects of the present disclosure areplayers, coaches, referees, or a combination thereof that are associatedwith a present game.

In some embodiments, each time-stamped position in an independentplurality of time-stamped positions for a respective player of the firstor second plurality of players includes an xyz-coordinate of therespective player with respect to the spatial region. For instance, insome embodiments the spatial region is mapped such that a center portionof the spatial region (e.g., half court, 50-yard line, etc.) is anorigin of an axis and a boundary region of the spatial region (e.g., anout of bounds line) is a maximum or minimum coordinate of an axis. Insome embodiments, the xyz-coordinate has an accuracy of ±5 centimeters,±7.5 centimeters, ±10 centimeters.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. A system, comprising: a communication interfaceconfigured to: receive a transmission of a central video feed from afirst camera; and receive a respective time-stamped position informationfrom a tracking system; and a processor coupled to the communicationinterface and configured to: calibrate the central video feed against aspatial region encompassed by the central video feed including by:partitioning at least one frame of the central video feed into aplurality of tiles at a first resolution; and determining a homographydefining placement of augmented reality elements on the at least oneframe of the central video feed; use the received time-stamped positioninformation and the plurality of tiles associated with at least oneframe of the central video feed to define a first sub-view of thecentral video feed, wherein the first sub-view corresponds to a portionof the spatial region encompassed by the central video feed; and outputthe first sub-view and the determined homography to a device configuredto use the first sub-view and the homography to display the firstsub-view.
 2. The system of claim 1, wherein: the first sub-view isassociated with a first subject of a plurality of subjects in thespatial region; and the first subject of the plurality of subjects inthe spatial region includes at least one player or object of interest.3. The system of claim 1, wherein the first sub-view includes aplurality of subjects.
 4. The system of claim 1, wherein: the firstsub-view includes a subset of the plurality of tiles associated with atleast one frame of the central video feed; and the subset of theplurality of tiles is selected based at least in part on a first subjectassociated with the first sub-view.
 5. The system of claim 1, whereinthe central video feed includes a full view of the spatial region andthe spatial region includes a competition space.
 6. The system of claim1, wherein the first sub-view of the central video feed includes anisolation shot of a portion of the spatial region.
 7. The system ofclaim 1, wherein the first sub-view includes a plurality of video framesdepicting a first subject.
 8. The system of claim 1, wherein the firstsub-view includes a plurality of video frames centered on a firstsubject.
 9. The system of claim 1, wherein the first sub-view iszoomable and a level of zoom is based at least in part on paddingsurrounding a first subject.
 10. The system of claim 1, whereinpartitioning at least one frame of the central video feed into aplurality of tiles at a first resolution includes dividing the at leastone frame into a predetermined number of tiles based at least in part ona desired resolution.
 11. The system of claim 1, wherein the augmentedreality elements include a scrimmage line.
 12. The system of claim 1,wherein the augmented reality elements include statistics associatedwith a competition corresponding to the central video feed.
 13. Thesystem of claim 1, wherein the augmented reality elements includescontent customized to a user.
 14. The system of claim 1, wherein theprocessor is configured to: capture a blank scene; and determineaugmented reality elements to add to the scene.
 15. The system of claim1, wherein the device is configured to use the first sub-view and thehomography to display the first sub-view including by reconstructing animage by compositing a blank scene, the augmented reality elements, anda first subject.
 16. The system of claim 1, wherein the first camera isfixed.
 17. The system of claim 1, wherein the first camera captures thecentral video feed at a resolution of at least 12K.
 18. The system ofclaim 1, wherein the device is configured to use the first sub-view andthe homography to display the first sub-view including by displaying agraphical user interface including a plays panel, a video panelincluding the first sub-view, and a player panel.
 19. A method,comprising: receiving a transmission of a central video feed from afirst camera; receiving a respective time-stamped position informationfrom a tracking system; calibrating the central video feed against aspatial region encompassed by the central video feed including by:partitioning at least one frame of the central video feed into aplurality of tiles at a first resolution; and determining a homographydefining placement of augmented reality elements on the at least oneframe of the central video feed; using the received time-stampedposition information and the plurality of tiles associated with at leastone frame of the central video feed to define a first sub-view of thecentral video feed, wherein the first sub-view corresponds to a portionof the spatial region encompassed by the central video feed; andoutputting the first sub-view and the determined homography to a deviceconfigured to use the first sub-view and the homography to display thefirst sub-view.
 20. A computer program product embodied in anon-transitory computer readable medium and comprising computerinstructions for: receiving a transmission of a central video feed froma first camera; receiving a respective time-stamped position informationfrom a tracking system; calibrating the central video feed against aspatial region encompassed by the central video feed including by:partitioning at least one frame of the central video feed into aplurality of tiles at a first resolution; and determining a homographydefining placement of augmented reality elements on the at least oneframe of the central video feed; using the received time-stampedposition information and the plurality of tiles associated with at leastone frame of the central video feed to define a first sub-view of thecentral video feed, wherein the first sub-view corresponds to a portionof the spatial region encompassed by the central video feed; andoutputting the first sub-view and the determined homography to a deviceconfigured to use the first sub-view and the homography to display thefirst sub-view.